The success of the MPEG-1 and MPEG-2 coding standards was driven by the fact that they allow digital audiovisual services with high quality and compression efficiency. However, the scope of these two standards is restricted to the ability of representing audiovisual information similar to analog systems where the video is limited to a sequence of rectangular frames. MPEG-4 (ISO/IEC JTC1/SC29/WG11) is the first international standard designed for true multimedia communication, and its goal is to provide a new kind of standardization that will support the evolution of information technology.
MPEG-4 provides for a unified audiovisual representation framework. In this representation, a scene is described as a composition of arbitrarilyy shaped audiovisual objects (AVOs). These AVOs can be organized in a hierarchical fashion, and in addition to providing support for coding individual objects, MPEGA also provides facilities to compose that hierarchical structure.
One of these AVOs is the Face Object, which allows animation of synthetic faces, sometimes called Talking Heads. It consists of a 2D representation of a 3D synthetic visual object representing a human face, a synthetic audio object, and some additional information required for the animation of the face. Such a scene can be defined using the BInary Format for Scene (BIFS), which is a language that allows composition of 2D and 3D objects, as well as animation of the objects and their properties.
The face model is defined by BIFS through the use of nodes. The Face Animation Parameter node (FAP) defines the part of the face has to be animated. The Face Description Parameter node (FDP) defines the rules to animate the face model. The audio object can be natural audio, or created at the decoder with some proprietary Text-To-Speech (TTS) synthesize. In the case of an encoded stream containing natural audio, an independent FAP stream drives the animation, and time stamps included in the streams enable the synchronization between the audio and the animation. A synthesizer is a device that creates an output based on a set of inputs and a set of rules. Two different synthesizers that are subjected to different rules may generate perfectly acceptable but markedly different outputs in the response to a given set of inputs, such as one synthesizer might generate a talking head of a blond woman, while the other might generate a talking head of a dark haired woman.
A TTS is a system that accepts text as input, and outputs an intermediate signal that comprises phonemes, and a final signal that comprises audio samples corresponding to the text. MPEG-4 does not standardize the TTS Synthesizer, but it provides a Text-To-Speech Interface (TTSI). By sending text to the decoder, the animation is driven by the FAP stream and by the TTS.
MPEG-4 defines a set of 68 Face Animation Parameters (FAPs), each corresponding to a particular facial action that deforms a face from its neutral state. These FAPs are based on the study of minimal perceptible actions, and are closely related to muscle action. The value for a particular FAP indicates the magnitude of the corresponding action. The 68 parameters are categorized into 10 groups, as shown in Table 1 of the appendix. Other than the first group, all groups are related to different parts of the face. The first group contains two high-level parameters (FAP 1 and FAP2) visemes and expressions. A viseme is a visual version of a phoneme. It describes the visually distinguishable speech posture involving the lips, teeth and tongue. Different phonemes are pronounced with a very similar posture of the mouth, like “p” and “b” and, therefore, a single viseme can be related to more than one phoneme. Table 2 in the appendix shows the relation between visemes and their corresponding phonemes.
In order to allow the visualization of mouth movement produced by coarticulation, transitions from one viseme to the next are defined by blending the two visemes with a weighting factor that changes with time along some selected trajectory.
The expression parameter (FAP 2) defines 6 high level facial expressions, such as joy, sadness, anger, etc. They are described in Table 3 of the appendix. The nine other FAP groups, which represent FAP 3 to FAP 68, are low-level parameters, like move left mouth corner up.
Each FAP (except FAP1 and FAP2) is defined in a unit, which can vary from one parameter to another. Unlike visemes and expressions, each low-level FAP characterizes only a single action. Therefore, a low-level action is completely defined with only two numbers, the FAP number, and the amplitude of the action to apply. In the case of high-level parameters, a third number, called FAPselect, is required to determine which viseme (in case of FAP 1), or which expression (in case of FAP 2) is to be applied.
For each frame, the receiver applies and performs the deformations on the face model using all FAPs. Once all actions have been done on the model, the face is rendered.
MPEG-4 allows the receiver to use a proprietary face model with its own animation rules. Thus, the encoder sends signals to control the animation of the face by sending FAPs but has no knowledge concerning the size and proportion of the head to animate, or any other characteristic of the decoding arrangements. The decoder, for its part, needs to interpret the values of they in a way such that the FAPs produce reasonable deformation. Because the encoder is not aware of the decoder that will be employed, the MPEG-4 standard contemplates providing normalized FAP values in face animation parameter units (FAPU). The FAPU are computed from spatial distances between key facial features on the model in its neutral state, such as iris diameter, eye separation, eye-to-nose separation, Mouth-to-nose separation, and Mouth width.
FIG. 1 presents a block diagram of a prior art face rendering arrangement that employs the FAP information that is available with MPEG-4. It includes an audio signal on line 10 that is applied to decoder 100, and a FAP stream on line 11 that is applied to face rendering module (FRM) 110. Module 110 is conventional. It can be a separate piece of hardware, but often it is a software module that is executed on a processor. A face model and its animation rules may be applied to FRM 110 via line 12. While decoder 100 decodes the audio signal, FRM 110 concurrently renders the face based on the applied FAP stream. Compositor 130, responsive to synthesizer 120 and FRM 110, simultaneously plays the audio and the animated model video that result from applying the FAPs to FRM 110. Synchronization is achieved at the decoder by retrieving timing information from the streams. This timing information is of two types, and must be included in the transmitted streams. The first type is used to convey the speed of the encoder clock, while the second one consists of time stamps attached to portions of the encoded data.
Providing for this synchronization (between what is said and the desired facial expressions) on the encoder side is not trivial, and the problem is certainly not reduced when a TTS arrangement is contemplated. The reason lies in the fact that whereas faces are animated at constant frame rate, the timing behavior of a TTS Synthesizer on the decoder side is usually unknown. It is expected that there will be a very large number of commercial applications where it will be desirable to drive the animation from a text. Therefore, solving the synchronization problem is quite important.